Repository navigation
Filter modules before loading them and load the kept ones in one call - #2283
Merged
josevalim merged 2 commits intoNov 2, 2026
Merged
Conversation
The retriever ran filter_modules only after a module had been loaded and fully processed (abstract code, specs, callbacks, markdown), so on a large project an allow-list of a few modules cost as much as documenting everything. Autolink then checked every module reference against the list of filtered module nodes with Enum.any?, which made a filtered build slower than an unfiltered one. Now the retriever applies filter_modules to the docs chunk alone, before Code.ensure_loaded and module_data, and returns the filtered modules as names. Autolink keeps them in a MapSet. The kept modules are loaded with one Code.ensure_all_loaded/1 call instead of a Code.ensure_loaded/1 round trip per module from parallel tasks, which serializes on the code server. Measured on a project with ~50k modules (6 schedulers): a 100-module allow-list went from 55-62 s to 22 s in the retriever+formatter phase with the modules preloaded, and from 468 s to 20 s without a preload; the full unfiltered build went from 473 s to 102 s without a preload. Output is unchanged. Co-Authored-By: Claude Fable 5.1 <[email protected]>
Contributor
Author
|
Hi @josevalim , when you have time, could you please take a look? |
|
📦 Docs artifacts are ready: https://github.com/elixir-lang/ex_doc/actions/runs/41499604864/artifacts/13167153751 |
josevalim
reviewed
Nov 2, 2026
Member
|
💚 💙 💜 💛 ❤️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two changes in
ExDoc.Retriever, one inExDoc.Autolink:filter_modulesruns on the docs chunk, before the module is loaded and processed. Previously every module went throughCode.ensure_loaded, abstract code translation, specs, callbacks and markdown parsing, and only then the predicate decided whether to keep the node. A filtered-out module now costs aCode.fetch_docsread and itsRefsentries, nothing else. The predicate receives the chunk's metadata, which is what the:filter_modulesdocs describe ("the module and its metadata"); it no longer sees the:kindkey the retriever adds to the node later.Code.ensure_all_loaded/1call.Code.ensure_loaded/1per module from sixTask.async_streamworkers serializes on the code server; on a large project that was most of the retrieval time.MapSetand checksMapSet.member?instead ofEnum.any?over a list ofModuleNodestructs for every module reference. The retriever returns the filtered modules as sorted names;ExDoc.Formatter.autolink/5builds the set once.Output is unchanged: the same pages, sidebar and search data, byte for byte apart from Makeup's per-run group prefix.
Why
On a private Elixir project with ~50 000 modules (~39 600 documented), measured locally with 6 schedulers:
filter_modulesallow-list of 100 modules, modules already loaded by an earlier stepmix docsalone)The 468 s and 473 s cases are the per-module
Code.ensure_loadedround trips; the project only avoided them by accident because an earlier task in the same VM had loaded every module. The allow-list case was additionally slowed down by theEnum.any?scan: with ~6 500 filtered modules a filtered build took 3.7× longer than an unfiltered one.Notes for review
Code.ensure_all_loaded/1exists since Elixir 1.15, the minimummix.exsrequires.docs_from_dir/2anddocs_from_modules/2now return{[ModuleNode.t()], [module()]}. Both are@doc false-level internals of a@moduledoc falsemodule; the existing tests that matched{nodes, []}still pass unchanged. Thefiltered_modulesautolink option is now aMapSetof names (tests updated).filter_modulesnow also asserts that the filtered module is never loaded.mix test: 465 passed.mix format --check-formatted: clean.🤖 Generated with Claude Code